Zero-Variance Importance Sampling Estimators for Markov Process Expectations

We study the structure of zero-variance importance sampling estimators for expectations of functionals of Markov processes. For a class of expectations that can be characterized as solutions to linear systems, we show that a zerovariance estimator can be constructed by using an importance distribution that preserves the Markovian nature of the underlying process. This suggests that good practical importance sampling distributions can be found by searching within the class of Markovian probability distributions. The class of expectations considered includes as particular cases, among others: the mean time until hitting a rare set, the expected cumulative discounted reward until hitting a set, the mean duration of an excursion, the transient (deterministic horizon) expectation of a discounted final payoff, and steady-state expectations.